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[Document Name] SPECIFICATION 
[Title of the Invention] 

Photographing Method, Learning Apparatus and Photographing Apparatus 
[Claims] 

[Claim 1 ] A photographing method for use in an image processing method in 
which an interpolation process is performed on an image signal output from a CCD 
element having a color filter array, thereby to generate an output image, said 
photographing method comprising: 

a class-tap extracting step of extracting a pixel as a class tap, from the input 
image, said pixel corresponding to various kinds of color signals; 

an encoding step of encoding the data of the class tap for each kind of a 
color signal; 

a classifying step of generating a class number representing a class defined 
on the basis of the result obtained in the encoding step and the relation among the 
color signals; 

a storing step of storing prediction coefficients predetermined for the class 
and outputting the prediction coefficient corresponding to the output of the 
classifying step; 

a prediction-tap extracting step of extracting a pixel at a prescribed position, 
as a prediction tap, from the input image; and 

a step of generating an output image by performing a prescribed operation 
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on the prediction coefficient output in the storing step and the data of the 
prediction tap. 

[Claim 2] The photographing method according to claim 1, wherein the 
encoding step includes a step of performing ADRC. 

[Claim 3] A learning apparatus for use in a photographing method which in 
turn is designed for use in an image processing method, wherein an interpolation 
process is performed on an image signal output from a CCD element having a 
color filter array, thereby to generate an output image, and which comprises: 

a class-tap extracting step of extracting a pixel as a class tap, from the input 
image, said pixel corresponding to various kinds of color signals; 

an encoding step of encoding the data of the class tap for each kind of a 
color signal; 

a classifying step of generating a class number representing a class defined 
on the basis of the result obtained in the encoding step and the relation among the 
color signals; 

a storing step of storing prediction coefficients predetermined for the class 
and outputting the prediction coefficient corresponding to the output of the 
classifying step; 

a prediction-tap extracting step of extracting a pixel at a prescribed position, 
as a prediction tap, from the input image; and 

a step of generating an output image by performing a prescribed operation 
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on the prediction coefficient output in the storing step and the data of the 
prediction tap, said learning apparatus comprising: 

image-converting means for receiving an image of the same signal format 
as the output image and generating an image signal a CCD element outputs, from 
the image of the same signal format as the output image; 

first image-extracting means for extracting a prescribed image region from 
the output of the image-converting means; 

classifying means for generating a class number representing a class output 
by the first image-extracting means; 

second image-extracting means for extracting a prescribed image region 
from the image of the same signal format as the output image; and 

means for generating the prediction coefficient from the output of the first 
image-extracting means, the output of the second image-extracting means and the 
output of the classifying means. 

[Claim 4] A photographing apparatus for use in an image processing 
apparatus in which an interpolation process is performed on an image signal output 
from a CCD element having a color filter array, thereby to generate an output 
image, said photographing apparatus comprising: 

class-tap extracting means for extracting a pixel as a class tap, from the 
input image, said pixel corresponding to various kinds of color signals; 

encoding means for encoding the data of the class tap for each kind of a 
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color signal; 

classifying means for generating a class number representing a class 
defined on the basis of the result obtained in the encoding step and the relation 
among the color signals; 

storage means for storing prediction coefficients predetermined for the 
class and outputting the prediction coefficient corresponding to the output of the 
classifying means; 

prediction-tap extracting means for extracting a pixel at a prescribed 
position, as a prediction tap, from the input image; and 

operation means for performing a prescribed operation on the prediction 
coefficient output by the storage means and the data of the prediction tap, thereby 
generating an output image, wherein the prediction coefficients are generated by 
prediction-coefficient generating means which has: 

image-converting means for receiving an image of the same signal format 
as the output image and generating an image signal a CCD element outputs, from 
the image of the same signal format as the output image; 

first image-extracting means for extracting a prescribed image region from 
the output of the image-converting means; 

classifying means for generating a class number representing a class output 
by the first image-extracting means; 

second image-extracting means for extracting a prescribed image region 
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from the image of the same signal format as the output image; and 

means for generating the prediction coefficient from the output of the first 
image-extracting means, the output of the second image-extracting means and the 
output of the classifying means. 
[Detailed Description of the Invention] 
[0001] 

[Technical Field of the Invention] 

The present invention relates to a photographing method, learning 
apparatus and photographing apparatus for use in video apparatuses such as digital 
cameras, which are designed to convert the output of a single-plate CCD to an 
image equivalent to one provided by a three-plate CCD. 

[0002] 

[Prior Art] 

Recently, compact digital cameras and VTRs each incorporating a camera 
have been put to practical use in great numbers, thanks to the advancement in 
electronic image technology and the miniaturization of electronic circuits. These 
photographing apparatuses are easy to operate to take pictures and reproducing 
images. Along with the increasing use of personal computers, they have come 
into common use. 
[0003] 

The photographing apparatus of this type comprise, for example, a CCD 
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(Charge Coupled Device) element each. The CCD element generates a signal 
charge output, which is supplied to an image-signal processing circuit through an 
AGC (Automatic Gain Control) circuit or the like. The signal charge output is 
subjected to gamma correction and white-balancing. Further, it is subjected to 
matrix conversion, whereby an image signal is generated. The image signal is 
recorded on a recording medium such as a floppy disk. 
[0004] 

Generally, conventional photographing apparatuses of this type incorporate 
a single-plate CCD element, so that it may be light. A color filter array having a 
specific color coating is provided in front of the single-plate CCD element. The 
image of an object of photography is applied to the CCD element through the color 
filter array. 
[0005] 

FIG. 12A shows an example of the color filter array. The color 
arrangement of the color- filter array, which is shown in FIG. 12A, is called "Bayer 
arrangement. " In this case, G-color filters are arranged in the pattern of a 
chessboard, and R-color filters and B-color filters are arranged in alternate 
columns, each in a vacant square. Each of the pixels of a single-plate CCD has 
only one data item representing R, G or B if primary-color filters are used. If 
complement-color filters are used, each pixel has only one data item representing 
Ye, Cy, Mg or G. Therefore, in processing the signal, interpolation is carried out 
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(see FIG. 12B) so that each pixel may have all signal components R, G and B or all 
signal components Ye, Cy, Mg and G. Hitherto known as an interpolation 
method is linear interpolation. 

[0006] 
[Obj ect of the Invention] 

When linear interpolation is carried, however, the waveform of the image 
is inevitably deformed, rendering the image unclear as a whole. A process such 
as edge emphasis must therefore be performed to increase the apparent resolution 
of the image. 

[0007] 

It is proposed that the classification-adaptation process, i.e., a process other 
than linear interpolation, be performed on the CCD outputs of the single-lens 
camera, for each of the R, G and B image signals, thereby to generate image 
signals that are equivalent to the CCD outputs of a three-plate camera. The 
classification-adaptation process is effected on the R, G and B image signals 
independently. Therefore, the same process is performed on each R pixel and 
each B pixel as on each G pixel (two existing in every four pixels), though only 
one R pixel exists in very four pixels and only one B pixel exists in very four 
pixels in the (m X n) pixels as shown in FIGS. 12A and 12B in the case where a 
color-filter array of Bayer arrangement is used. Consequently, high-precision 
prediction cannot be accomplished as far as the R-component signals and 
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B-component image signals are concerned. 
[0008] 

Accordingly, the object of this invention is to provide a photographing 
method, a learning apparatus and a photographing apparatus, which can perform an 
interpolation process with high accuracy and which can generate image data of 
high resolution. 
[0009] 

[Means to Solve the Problem] 

The invention described in claim 1 is a photographing method for use in an 
image processing method in which an interpolation process is performed on an 
image signal output from a CCD element having a color filter array, thereby to 
generate an output image. The photographing method is comprises: 

a class-tap extracting step of extracting a pixel as a class tap, from the input 
image, said pixel corresponding to various kinds of color signals; 

an encoding step of encoding the data of the class tap for each kind of a 
color signal; 

a classifying step of generating a class number representing a class defined 
on the basis of the result obtained in the encoding step and the relation among the 
color signals; 

a storing step of storing prediction coefficients predetermined for the class 
and outputting the prediction coefficient corresponding to the output of the 
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classifying step; 

a prediction-tap extracting step of extracting a pixel at a prescribed position, 
as a prediction tap, from the input image; and 

a step of generating an output image by performing a prescribed operation 
on the prediction coefficient output in the storing step and the data of the 
prediction tap. 
[0010] 

The invention defined in claim 3 is a learning apparatus for use in a 
photographing method which in turn is designed for use in an image processing 
method, wherein an interpolation process is performed on an image signal output 
from a CCD element having a color filter array, thereby to generate an output 
image. The photographing method comprises: 

a class-tap extracting step of extracting a pixel as a class tap, from the input 
image, said pixel corresponding to various kinds of color signals; 

an encoding step of encoding the data of the class tap for each kind of a 
color signal; 

a classifying step of generating a class number representing a class defined 
on the basis of the result obtained in the encoding step and the relation among the 
color signals; 

a storing step of storing prediction coefficients predetermined for the class 
and outputting the prediction coefficient corresponding to the output of the 
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classifying step; 

a prediction-tap extracting step of extracting a pixel at a prescribed position, 
as a prediction tap, from the input image; and 

a step of generating an output image by performing a prescribed operation 
on the prediction coefficient output in the storing step and the data of the 
prediction tap. 

The learning apparatus is characterized by comprising: 

image-converting means for receiving an image of the same signal format 
as the output image and generating an image signal a CCD element outputs, from 
the image of the same signal format as the output image; 

first image-extracting means for extracting a prescribed image region from 
the output of the image-converting means; 

classifying means for generating a class number representing a class output 
by the first image-extracting means; 

second image-extracting means for extracting a prescribed image region 
from the image of the same signal format as the output image; and 

means for generating the prediction coefficient from the output of the first 
image-extracting means, the output of the second image-extracting means and the 
output of the classifying means. 
[0011] 

The invention described in claim 4 is a photographing apparatus for use in 
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an image processing apparatus in which an interpolation process is performed on 
an image signal output from a CCD element having a color filter array, thereby to 
generate an output image. The photographing apparatus comprising: 

class-tap extracting means for extracting a pixel as a class tap, from the 
input image, said pixel corresponding to various kinds of color signals; 

encoding means for encoding the data of the class tap for each kind of a 
color signal; 

classifying means for generating a class number representing a class 
defined on the basis of the result obtained in the encoding step and the relation 
among the color signals; 

storage means for storing prediction coefficients predetermined for the 
class and outputting the prediction coefficient corresponding to the output of the 
classifying means; 

prediction-tap extracting means for extracting a pixel at a prescribed 
position, as a prediction tap, from the input image; and 

operation means for performing a prescribed operation on the prediction 
coefficient output by the storage means and the data of the prediction tap, thereby 
generating an output image. 

The prediction coefficients are generated by prediction-coefficient 
generating means. The prediction-coefficient mans has: 

image-converting means for receiving an image of the same signal format 
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as the output image and generating an image signal a CCD element outputs, from 
the image of the same signal format as the output image; 

first image-extracting means for extracting a prescribed image region from 
the output of the image-converting means; 

classifying means for generating a class number representing a class output 
by the first image-extracting means; 

second image-extracting means for extracting a prescribed image region 
from the image of the same signal format as the output image; and 

means for generating the prediction coefficient from the output of the first 
image-extracting means, the output of the second image-extracting means and the 
output of the classifying means. 
[0012] 

In each invention described above, an ADRC process is performed on the 
data of a class tap, which contains color signals, for each color signal from the 
color filter array. A class is determined from the relation the result of the ADRC 
process and the color signal. 
[0013] 

[Preferred Embodiment of the Invention] 

The present invention will be described, with reference to an embodiment 
that is a digital still camera. FIG. 1 shows the embodiment in its entirety. The 
lens section 1 has a zooming mechanism, focusing mechanism, iris mechanism and 
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the like. As will be described later, automatic diaphragm control and automatic 
focus (AF) control are effected in accordance with the control data supplied from a 
main CPU (microcomputer) 10. Light is applied via the lens section 1 in a 
prescribed amount, thus forming an image of an object of photography on a CCD 
2. 
[0014] 

On the pixels of the CCD 2, a color filter array is provided which has a 
color coating. The CCD 2 is exposed to light for a predetermined time, as it is 
controlled by a timing signal supplied from a timing generating circuit 6. The 
CCD 2 converts the light applied to it via the color filters to a signal charge (analog 
data). The signal charge output by the CCD 2 is supplied to an AGC (Automatic 
Gain Control)/CDS (Correlated Double Sampling) circuit 3. The AGC/CDS 
circuit 3 adjusts the output of the CCD 2 to a constant level and removes the 1/f 
noise generated in the output circuit of the CCD 2. 
[0015] 

The timing generating circuit 6 supplies a timing signal to the AGC/CDS 
circuit 3. The timing signal controls the electronic shutter of the camera. The 
output of the AGC/CDS circuit 3 is supplied to an A/D converter 4. The A/D 
converter 4 receives a signal from the timing generating circuit 6. This signal 
represents sampling timing. In accordance with the signal the A/D converter 4 
performs sampling, generating a digital image signal. For example, the converter 
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4 generates 10 bits at each sampling. The digital image signal is supplied to an 
image-signal processing section 5. 
[0016] 

The image-signal processing section 5 performs defect-correcting process, 
digital clamping process, white-balance adjusting process, gamma-correcting 
process, interpolation using classification adaptation, and the like, on the image 
signal input to it. A memory 12 is connected to the image-signal processing 
section 5. The data being processed may be stored into the memory 12, if 
necessary. The image data generated by the image-signal processing section 5 is 
supplied to a storage-medium interface (I/F) 13. The timing generating circuit 6 
generates various timing signals. The timing signals are supplied to the CCD 2, 
AGC/CDS circuit 3, A/D converter 4, main CPU 10 and the like. The circuits are 
thereby operated in synchronism with one another. 
[0017] 

The motor 7 drives the iris mechanism of the lens section 1 in accordance 
with the control data supplied from the main CPU 10. Thus driven, the iris 
mechanism controls the amount of incident light passing through the lens. The 
motor 8 moves the lens in accordance with the control data supplied from the main 
CPU 10, thus controlling the focusing condition. Automatic iris control, 
automatic focus (AF) control and the like can thereby accomplished. A flash 
section 9 is provided to emit flashing light to the object in a predetermined amount, 
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under the control of the main CPU 10. 
[0018] 

The storage-medium interface (I/F) 1 3 supplies the image data output from 
the image-signal processing section 5, into a memory 14, if necessary. The 
interface 1 3 also effects a prescribed interface process on the image data and then 
supplies the data to a storage medium 15, whereby the data is stored into the 
medium 15. The storage medium 15 is one that can be removably set in the main 
body of the digital still camera. It may be a floppy disk, a disk-shaped recording 
medium for use in hard disk drives, a flash memory such as a memory card, a 
memory stick or the like. 
[0019] 

A controller 11 is provided, which is controlled by the CPU 10 to supply 
control data to the image-signal processing section 5 and the storage-medium 
interface 13, thereby controlling the section 5 and the interface 13. A signal 
indicating an operation is supplied to the CPU 10 via a terminal 17 when the user 
operates an operation input section (not shown). A power-supply section 16 is 
provided, which has a battery 16a, a DC/DC converter 16b and the like. The 
DC/DC converter 16b converts the power supplied from the battery 16a to a DC 
voltage, which is applied to the components of the camera. The battery 1 6a is a 
rechargeable one and can be removed from the main body of the digital camera. 
[0020] 
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With reference to FIG. 2 it will be described how an object of photography 
is photographed and the data representing the image of the object is stored into the 
storage medium 15. In Step Sl 5 the lens of the lens section 1 collects the light 
reflected from the object. The light is applied through the iris to the CCD 2. In 
Step S2, it is determined whether the user has operated the release button provided 
on the camera. If it is determined in Step S2 that the release button has been 
operated, the operation goes to Step S3. Otherwise, the operation returns to Step 
SI. Hence, Steps SI and S2 are repeated until the user operates the release button. 
In Step S3, the CCD 2 converts the light reflected from the object to an electric 
signal. 
[0021] 

In Step S4, the level of the signal charge generated in Step S3 is adjusted 
(AGC process), and the 1/f noise is removed from the signal charge (CDS process). 
In Step S5, A/D conversion is carried out, generating a digital image signal, or 
generating ten bits at each sampling. In Step S6, the digital image signal 
generated in Step S5 is processed, by means of classification-adaptation process 
and the like. In Step S7, the image signal obtained in Step S6 is subjected to 
interface process and then stored into the storage medium. 
[0022] 

The image-signal processing section 5 will be described, with reference to 
FIG. 3. The digital image signal is supplied from the A/D converter 4 shown in 
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FIG. 1 to a defect-correcting circuit 21. The defect-correcting circuit 21 performs 
the process of eliminating a defective part of the image signal, which corresponds 
to a pixel or pixels of the CCD 2 that fail to respond to the input light or to a pixel 
or pixels that always hold an electric charge. The output of the defect-correcting 
circuit 21 is supplied to a clamping circuit 22. The clamping circuit 22 performs 
a process that is equivalent to the level-shifting process effected in the A/D 
converter 4. That is, the A/D converter 4 shifts the signal value a little in the 
positive direction, in order to prevent the negative value from being cut. On the 
other hand, the clamping circuit 22 shifts the signal back in the opposite direction. 
The negative value of the signal level is thereby maintained. 
[0023] 

The output of the clamping circuit 22 is supplied to a white-balancing 
circuit 23. The white-balancing circuit 23 corrects the gains of the R, G and B 
signals input to it. The output of the white-balancing circuit 23 is supplied to a 
gamma-correcting circuit 24. The gamma-correcting circuit 24 corrects the value 
of the signal input from the white-balancing circuit 23, in accordance with a 
gamma curve. An interpolating section 31 is provided, which performs a 
classification-adaptation process, thereby converting the output of the 
gamma-correcting circuit 24 to an image signal that is equivalent to one generated 
by a three-plate CCD. 
[0024] 
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A correction circuit 29 is provided, which performs correction, such as 
edge correction (so-called "image forming"), on the output of the interpolating 
section 3 1 . The output of the correction circuit 29 is supplied to an RGB matrix 
circuit 30. The RGB matrix circuit 30 converts the RGB signal supplied from the 
correction circuit 26, to image data of a prescribed format, such as YUV. 
Alternatively, the matrix conversion may not be performed, and the RGB signal 
may be output from the RGB matrix circuit 30 without being processed at all. In 
the present embodiment, the user can select either the YUV signal or the RGB 
signal. 
[0025] 

The interpolating section 31 will be described in detail. To facilitate 
understanding, ordinary classification-adaptation process will be first explained. 
FIG. 4 shows an example of a device that performs prediction operation by means 
of classification-adaptation process. The input image is supplied to 
region-extracting sections 101 and 102. The region-extracting section 101 
extracts a specified region (called "class tap") from the input image. The data of 
the class tap is supplied to an ADRC (Adoptive Dynamic Range Coding) section 
103. The ADRC section 103 performs an ADRC process on the data supplied to 
it, thereby generating a re-quantized code. A method other than the ADRC 
process may be used to generate a re-quantized code. 
[0026] 
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The ADRC process will be explained. ADRC is an adaptive 
re-quantization method that has been developed to achieve high-efficiency 
encoding in VTRs (Video Tape Recorders). The method is advantageous in that a 
local signal-level pattern can be expressed in small amount of data. Thanks to 
this advantage, the method can be utilized to detect a pattern in the time-space of 
an image signal. The ADRC process section 103 re-quantizes the class tap, i.e., 
the region extracted. More precisely, it divides the difference between the 
maximum value MAX and minimum value MIN in the class tap, by the number of 
bits designated, in accordance with the following equations: 
[0027] 

DR - MAX - MIN+1 

Q = [(L - MIN + 0.5) X 2 n /DR] ...(1) 
where DR is the dynamic range of the region and n is the number of bits allocated 
to the region, L is the signal level of pixels present in the region, and Q is the 
re-quantized code. For example, n may be 2, that is, n = 2. The bracket ([....]) 
means the process of omitting the decimal fractions. 
[0028] 

Thus, an image-signal class tap in which each pixel consists of, for 
example, eight bits is converted to a 2-bit re-quantized code. The re-quantized 
code, thus generated, represents the level-distribution pattern in the image-signal 
class tap, by 
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using a small amount of information. If the class tap is composed of, for example, 
seven pixels, the above-mentioned process is performed, thereby generating seven 
re-quantized codes qi to q-j. The class code is of such a type as given by the 
following equation (2): 

[0029] 
[Math 1] 

n 

class = ^jJi(2 p ) 1 

... (2) 

[0030] 

where n is the number of pixels to be extracted as a class tap. The value for p 
may be 2, that is, p = 2. 
[0031] 

The class code, class, is characteristic data that represents the space activity, 
i.e., the level-distribution pattern in the time-space of the image signal. The class 
code, class, is supplied to a prediction coefficient memory 104. The prediction 
coefficient memory 104 stores sets of prediction coefficients, each set assigned to 
one class as will be described later. The memory 104 outputs the set of prediction 
coefficients that corresponds to the class identified by the re-quantized code 
supplied to the memory 104. In the meantime, the region-extracting sections 102 
extracts a region of the image (called "predicted tap") and supplies the image 
signal of the predicted tap to a prediction section 105. The prediction section 105 
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performs the operation of the following equation (1) on the set of coefficients 
supplied from the prediction coefficient memory 104, thereby generating an output 
image signal y. 
[0032] 

y = wi X xi+w 2 X x 2 + .... + w n X x n "-(3) 
where x i5 and x n are the values of the pixels constituting the predicted tap, and 
Wi, and w n are the prediction coefficients. 
[0033] 

The process of determining prediction coefficients will be explained, with 
reference to FIG. 5. A HD (High-Definition) image signal having the same image 
format as the output image signal is supplied to a HD-SD converter section 201 
and a pixel-extracting section 208. The HD-SD converter section 201 carries out 
extraction or the like, whereby the HD image signal is converted to an image signal 
(hereinafter referred to as "SD (Standard-Definition) image signal") that has a 
resolution (i.e., number of pixels) similar to that of the input image signal. The 
SD image signal is supplied to region-extracting sections 202 and 203. Like the 
region-extracting section 101 shown in FIG. 4, the region-extracting section 202 
extracts a class tap from the SD image signal and generates an image signal 
representing the class tap. This signal is supplied to an ADRC process section 
204. 
[0034] 
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The ADRC process section 204 performs the same ADRC process as does 
the ADRC process section 103 shown in FIG. 4. That is, the section 204 
performs ADRC process, thus generating a re-quantized code from the signal 
supplied to it. The re-quantized code is supplied to a class-code generating 
section 205. The class-code generating section 205 is generates a class code that 
represents the class of the re-quantized code supplied to the section 205. The 
class code is supplied to a normal-equation adding section 206. Meanwhile, the 
region-extracting section 203 extracts a predicted tap from the SD image signal in 
the same way as does the region-extracting section 102 shown in FIG. 4. The 
image signal representing this predicted tap is supplied to the normal-equation 
adding section 206. 
[0035] 

The normal-equation adding section 206 adds the image signals supplied 
from the region-extracting sections 203 to the image signals supplied from a 
pixel-extracting section 208, for each class code supplied from the class-code 
generating section 205. The signals, thus added for each class code, are supplied 
to a prediction-coefficient determining section 207. Based on the signals for each 
class code, the prediction-coefficient determining section 207 determines 
prediction coefficients. 
[0036] 

The operation that is performed to determine a set of prediction coefficients 
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will be explained. Various types of image data items are supplied, as an HD 
image, to the device shown in FIG. 5. If the number of the image data items is m ? 
we obtain the following equation (4) from the equation (3): 
[0037] 

y k = w x X x kl +w 2 X Xk2 + + w n X Xkn ...(4) 

(k=l,2, m) 

If m > n, coefficients wi, and w n cannot be determined directly. Thus, 
the element e k of an error vector e is defined as shown in the following equation 
(5), thereby to determine prediction coefficients so that the square of the error 
vector e defined by the equation (6) presented below may become minimal. That 
is, so-called "least squares method" is applied, thereby directly determining 
prediction coefficients. 
[0038] 

e k = y k "{wi X x k i+w 2 X x^ + .^. + Wn X Xkn } ...(5) 
(k=l,2, m) 

[0039] 
[Math 2] 

m 

« 2 =5>* 2 - ( 6 ) 

[0040] 

A method that may actually used to obtain a set of prediction coefficients, 
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which minimizes the value e 2 in the equation (6), is to find partial differential 
values of e 2 for the prediction coefficients Wi (i = 1, 2, ....), as indicated by the 
following equation (7). In the method it suffices to determine each prediction 
coefficient Wi to impart a partial differential value of 0 to the prediction coefficient. 



The sequence of operations for determining the prediction coefficients Wj 
by using the equation (7) will be described. Let Xji and Yj be defined by the 
equations (8) and (9). The equation (7) can then be rewritten to the following 
equation (10): 

[0043] 
[Math 4] 



[0041] 



[Math 3] 




... (7) 



[0042] 



m 



PJ 



... (8) 



[0044] 



[Math 5] 



m 



k=0 



... (9) 
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[0045] 
[Math 6] 





X X2 . 
























x n2 . 


- X nn_ 









[0046] 

The equation (10) is generally called "normal equation." The 
normal-equation adding section 205 carries out operations of the equations (8) and 
(9) on the signal supplied to it, thereby calculating Xp and Yj (i = 1, 2, n). The 
prediction-coefficient determining section 207 solves the normal equation (10) by 
using an ordinary matrix analysis such as sweeping method, thereby calculating 
prediction coefficients wj (i = 1, 2, n). 
[0047] 

The classification-adaptation process described above is carried out, 
whereby various image-data conversion are effected, generating, from the input 
image, an image free of noise or an image with its scanning-line configuration 
changed. In the present invention, the input image, which is an image generated 
by a single-plate CCD, is converted to an image equivalent to one generated by a 
three-plate CCD. 
[0048] 
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Referring to FIG. 3 again, the adaptation process section 31 includes an 
ADRC (Adaptive Dynamic Range Coding) circuit 25 , a classification process 
circuit 26, an adaptation process circuit 27, a coefficient memory 28. The ADRC 
process circuit 25 extracts a class tap from the output of the gamma-correcting 
circuit 24. It performs ADRC process on the data of the class tap extracted, 
thereby generating a re-quantized code. The class tap includes pixels that 
correspond to color signals. The ADRC process is performed on pixels 
corresponding to each color signal. Nine pixels may be extracted as one class tap 
and four of these nine may correspond to R. In this case, the re-quantized code 
generated by ADRC-processing the four pixels is used when the 
classification-adaptation process is performed on R. 
[0049] 

The output of the ADRC circuit 25 is supplied, as characteristic data, to the 
classification circuit section 26. The classification process circuit 26 classifies 
the pattern of the image signal on the basis of the re-quantized code and the 
relation between the color signals. The circuit 26 generates a class number that 
represents the result of classification. Since the relation between the color signals 
is applied to the result of the classification can be effected more appropriately than 
otherwise. This enhances the precision of the interpolation process. 
[0050] 

The class number is supplied to the adaptation process circuit 27. The 
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adaptation process circuit 27 reads the prediction coefficients that correspond to 
the class number supplied to it, from the coefficient memory 28. The prediction 
coefficients stored in the coefficient memory 28 have been determined by a 
learning apparatus as will be described later. The adaptation process circuit 27 
extracts a prescribed pixel region, as a prediction tap, from the image signals. 
The circuit 27 carries out the operation of the following equation (1) on the data of 
the prediction tap and the prediction coefficients read as described above, thereby 
generating a predicted pixel value y. The output of the adaptation process circuit 
27 is supplied to the correction circuit 29, as the output of the adaptation process 
section 3 1 . 
[0051] 

How the prediction coefficients are acquired by learning will be explained, 
with reference to FIG. 6. The output image to be generated by the 
classification-adaptation process, i.e., an image of the same signal format as an 
image equivalent to one generated by a three-plate CCD, is supplied as a teacher 
image signal to an extraction circuit 41 and a teacher-image block circuit 45. The 
extraction circuit 41 extracts pixels from the teacher-image signal, in accordance 
with the arrangement of the color filters constituting an array. Pixels are 
extracted from the teacher-image signal by using filters equivalent to the optical 
low-pass filters, with respect to the CCD 2. That is, pixels are extracted in 
consideration of the optical system actually employed. The output of the 
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extraction circuit 41 is a student-image signal, which is supplied to a student- image 
block generating circuit 42. 
[0052] 

The student-image block generating circuit 42 extracts the class tap and 
prediction tap related to the pixel of interest, from the student-image signal 
generated by the extraction section 41 , while referring to the relation between the 
teacher-image signal and the predicted pixel for each block. At this time, the 
class tap is set at a signal value for the color filter array that has the densest data. 
The output of the student-image block generating circuit 42 is supplied to an 
ADRC process circuit 43. The ADRC process circuit 43 performs an ADRC 
process on the data of the class tap including the color signals extracted in the 
student-image block generating circuit 42, color signal by color signal. 
[0053] 

The output of the ADRC circuit 43 is supplied to a classification circuit 44. 
The classification circuit 44 determines a class from the output of the ADRC 
circuit 43 and the relation between the color signals. The class number 
corresponding to the class thus determined is supplied to an operating circuit 46. 
Meanwhile, the data of the prediction tap the student-image block generating 
circuit 42 has extracted is supplied to the operating circuit 46 through the ADRC 
circuit 43 and the classification circuit 44. 
[0054] 
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The teacher-image block generating circuit 45 extracts the predicted image 
from the teacher image, while referring to the class tap of the student image. The 
data of the predicted image is supplied to the operating circuit 46. The operating 
circuit 46 generates the data of the normal equation whose solution is the 
prediction coefficient, in accordance with the class number supplied from the 
classification circuit 44. In the process, the circuit 46 maintains the relation 
between the data of the prediction tap supplied from the classification circuit 44 
and the predicted image supplied from the teacher-image block generating circuit 
45. 
[0055] 

The data of the normal equation, generated by the operating section 46, is 
supplied to a learned data memory 47. The learned data memory 47 reads the 
data sequentially and store it. Another operating circuit 48 is provided, which 
solves a normal equation by using the data stored in the learned data memory 47. 
Prediction coefficients is thereby calculated for one class. The prediction 
coefficients, thus calculated, are stored into a coefficient memory 49. The 
contents of the coefficient memory 49 are loaded into the coefficient memory 28 
shown in FIG. 3, and will be utilized to achieve the classification-adaptation 
process. 
[0056] 

As described above, an image having the same signal format as an image 
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equivalent to one generated by a three-plate CCD is supplied as a teacher image, 
and an extraction process is performed on the teacher image, thereby generating a 
student image having the same signal format as an image equivalent to one 
generated by a single-plate CCD. Instead, an image having a higher data density 
than an image equivalent to one generated by a three-plate CCD may be input, and 
the extraction process may be performed on this image, thereby to generate a 
teacher image and a student image. 
[0057] 

The present invention can be applied, also to the case where an image 
different in solution from a CCD image, for example by increasing the pixels of 
each row and column of the CCD image sensor, thus enhancing the resolution four 
times. That is, a learning process may be effected, using the image signal to be 
generated as a teacher-image signal and the image signal output from the CCD 
incorporated in the digital still camera, thereby generating prediction coefficients. 
The prediction coefficients, thus generated, may then be applied to the 
classification-adaptation process. 
[0058] 

How the image signals are processed will be explained with reference to 
the flowchart of FIG. 7. In Step Sll, the defects in the CCD incorporated are 
detected, and the image signal is corrected in accordance with the defects detected. 
In Step S12, clamping is performed on the image signal corrected, thereby 
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eliminating that component of the image signal that has been shifted in the positive 
direction not to impair the negative value. In Step SI 3, white-balancing is 
effected, thereby adjusting the gains of color signals. In Step S14, the gamma 
correction is carried out. 
[0059] 

In Step SI 5, a classification- adaptation process is effected, thereby 
performing interpolation. This process consists of Steps SI 51 to SI 55. In Step 

5151, a block is generated by extracting a class tap and a prediction tap. The 
class tap contains pixels corresponding to various types of color signals. In Step 

51 52, an ADRC process is effected for different types of color signals. In Step 

51 53, a class is determined from the results of the ADRC process and the relation 
between the color signals. A class number corresponding to the class thus 
determined is output. In Step SI 8, the operation of the equation (1) is carried out 
on the prediction coefficient corresponding to the class number and the data of the 
prediction coefficient, thereby generating a predicted pixel value. In Step 155, it 
is determined whether or not the processes have been performed on the entire 
region. If it is determined that the processes have been performed on all regions, 
the operation goes to Step SI 6. Otherwise, the operation goes to Step SI 52 and 
the next region will be processed. 

[0060] 

In Step SI 6, a correction process (so-called "picture-forming process") is 
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effected on the image signal, which has been generated in Step S15 and which is 
comparable with one output by the CCD of a three-plate camera. The visual 
characteristics of the resultant image can thereby be improved. In Step SI 7, 
color-space conversion is performed on the image signal obtained in Step SI 6. 
For example, a RGB signal is converted to a YUV signal. An output image 
having a desirable signal format is thereby generated. 
[0061] 

The process of calculating prediction coefficients will be explained with 
reference to FIG. 8. In Step S31, an extraction process is performed on the 
teacher-image signal that represents an image comparable, in quality, with an 
image picked up by a three-plate CCD. A student-image signal equivalent to an 
output of a single-plate CCD is thereby generated. In Step S32, a block is 
generated, while maintaining the mutual relation between the student pixels and 
the teacher pixels. That is, a class tap and a prediction tap are extracted. In Step 
S33, the data of the class tap extracted from the student image is subjected to 
ADRC process, for each color signal. In Step S34, the relation between the color 
signals is added to the data of the ADRC-processed class tap that is the result of 
Step S33, thereby classifying a class. A signal is generated, which indicates the 
class number corresponding to the class classified. 
[0062] 

In Step S3 5, the data items of the prediction taps, each for one class 
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classified, are sequentially accumulated, thereby finding data for the normal 
equation. In Step S3 6, it is determined whether the data accumulation of Step 
S35 has been carried out on all blocks extracted from the image. If it is 
determined that the data accumulation has been effected on all blocks, the 
operation goes to Step S3 7. Otherwise, the operation returns to Step S3 5. In 
this case, the data accumulation is continued for the data items of the remaining 
blocks. In Step S37, the normal equation is solved. 
[0063] 

In Step S3 6, it is determined whether or not the normal equation has been 
solved for all classes, thereby determining the prediction coefficients of Step S3 5. 
If it is determined that the normal equation has been solved for all classes, the 
operation is terminated. Otherwise, the operation returns to Step S3 7. In this 
case, the operation is performed on the classes for which prediction coefficients 
have not been determined. The prediction coefficients thus obtained are supplied 
to the coefficient memory 49 and stored therein. The prediction coefficients 
stored in the coefficient memory 49 are loaded into the coefficient memory 28 
shown in FIG. 3. 
[0064] 

FIGS. 9 A to 9G show various arrangements of primary-color filter arrays 
that can be used in combination of the CCD 2 incorporated in the digital camera 
according to the invention. FIG. 9A shows a Bayer arrangement of color filters. 
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FIG. 9B depicts an interline arrangement. FIG. 9C illustrates a G-stripe ? RB 
chessboard arrangement. FIG. 9D depicts a G-stripe, RB perfect chessboard 
arrangement. FIG. 9E illustrates a stripe arrangement. FIG. 9F depicts a slant 
stripe arrangement. FIG. 9G illustrates a primary color-difference arrangement. 
[0065] 

FIGS. 9H to 9N illustrate various arrangements of color- filter arrays which 
may be used in combination with the CCD 2 incorporated in the digital camera 
according to the invention. FIG. 9H shows a field color-difference sequence 
arrangement. FIG. 91 depicts a frame color-difference sequence arrangement. 
FIG. 9J shows a MOS-type arrangement. FIG. 9K illustrates a modified 
MOS-type arrangement. FIG. 9L shows a frame interleave arrangement. FIG. 
9M depicts a field interleave arrangement. FIG. 9N depicts a strip arrangement. 
[0066] 

As mentioned above, various types of color filter arrays are available that 
can be used in combination with a single-plate CCD 2. The present invention is 
effective in every case where the data items the signals have in a color filter array 
differ in density from one another. 
[0067] 

Specific examples of the class tap and prediction tap applied to the image 
signal color-coded by the color-filter array of Bayer arrangement (shown in FIG. 
9A) used in combination with the CCD 2 will be described in detail, with reference 
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to FIGS. 10 and 11. FIG. 10 illustrates a tap that serves to generate an R signal, a 
G signal and a B signal at the position of a B pixel (an image signal having a B 
component, too, will be generated at the position of the B pixel). FIG. 1 1 shows 
that serves to generate an R signal, a G signal and a B signal at the position of a G 
pixel (an image signal having a B component, too, will be generated at the position 
of the B pixel). In FIGS. 10 and 1 1, the double circle indicates the position of a 
pixel to be predicted. (Hereinafter, such a pixel will be referred to as "predicted 
pixel.") In FIGS. 10 and 11, the squares represent the positions of pixels to be 
extracted to constitute a class tap, and the triangles indicate the positions of the 
pixels to be extracted to form a prediction tap. 
[0068] 

To generate an R signal, a G signal and a B signal at the position of a G 
pixel as is illustrated in FIG. 10A, the pixels specified in FIG. 10B are extracted 
and used as a class tap. That is, nine pixel (indicated by the triangles), i.e., the 
pixel to be predicted and the eight pixels surrounding the pixel to be predicted, are 
extracted. If the pixels corresponding to any color signal are insufficient in 
number, more pixels indicated by squares are extracted as a class tap. 
[0069] 

An ADRC process is performed on the signals of the class tap. Only the 
central pixel is extracted from the results of the ADRC process, thereby achieving 
classification. Furthermore, a process is effected to apply the relation between 
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the R, G and B pixels to the results of the classification. For example, the 
dynamic range of the ADRC process performed on each signal, the results of 
threshold process, the maximum and minimum dynamic ranges of the ADRC 
process, and the like are added to the image signal, in the form of data items 
consisting of several bits. An image signal can thereby predicted with high 
precision and generated, which represents a high-resolution image. 
[0070] 

The prediction tap is extracted as is shown in FIG. IOC. More 
specifically, the pixels at which R, G and B components will be mixed are 
extracted as a prediction tap. The prediction coefficients, provided for the class, 
are applied in weighting process, addition process and the like, thereby predicting 
an image signal at the position predicted. 
[0071] 

In the case of the Bayer arrangement, the number of pixels for each R 
signal, the number of pixels for each G signal and the number of pixels for each B 
signal are 1, 2 and 1, respectively. Hence, a class tap and prediction tap of the 
same structures as those applied to predict and generate the R, G and B signals at 
the position of the R pixel can be utilized to predict and generate an R signal, a G 
signal and a B signal at the position of the R pixel. 
[0072] 

To generate an R signal, a G signal and a B signal at the position of a G 
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pixel as is illustrated in FIG. 11 A, the pixels specified in FIG. 11B are extracted 
and used as a class tap. That is, nine pixel (indicated by the triangles), i.e., the 
pixel to be predicted and the eight pixels surrounding the pixel to be predicted, are 
extracted. For any color signal having a small number of pixels, the class tap is 
expanded to include those pixels that are indicated by squares. An ADRC 
process is performed on the data of the class tap. Only the central pixel is 
extracted from the results of the ADRC process, thereby achieving classification. 
Furthermore, a process is effected to apply the relation between the R, G and B 
pixels to the results of the classification, in the same way as in the process 
described with reference to FIG. 10. 
[0073] 

A prediction tap is extracted as shown in FIG. 11C. More precisely, the 
pixels at which R, G and B components will be mixed are arranged as a class tap. 
The set of prediction coefficients, provided for the class, are applied in weighting 
process, addition process and the like, thereby predicting an image signal at the 
position predicted. 
[0074] 

To evaluate the operating efficiency of the embodiment described above, 
simulation was conducted on the assumption that a color-filter array of the Bayer 
arrangement is utilized. In the simulation, prediction coefficients were generated 
by applying algorithm similar to the one that is used in the learning apparatus. 
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Further, an extraction process was carried out, generating an image signal 
equivalent to an output of a single-plate CCD, from an image signal equivalent to 
an output of a three-plate CCD. Still further, a prediction process was 
implemented in the above-mentioned classification-adaptation process. Moreover, 
simulation was conducted by means of linear interpolation and by means of the 
classification-adaptation process according to the invention. In the 
classification-adaptation process, R, G and B pixels were classified independently 
of one another. The results of the linear interpolation were compared with the 
results of the classification-adaptation process. 
[0075] 

The simulation was conducted on nine high-vision images of the ITE 
(Institute of Television Engineers) standard. The nine high- vision images were 
used, also to calculate a set of prediction coefficients. The simulation resulted in 
an image signal that was sharper at edges and fine parts than the image signal 
generated by the linear interpolation. In addition, it was confirmed that the S/N 
ratio had improved. Moreover, the resolution of any images represented by R- 
and B-component image signals was higher than in the case where the 
classification-adaptation process was effected on the R, G and B image signals 
independently. Thus, the embodiment of the present invention can provide 
images that are superior, in terms of the sharpness of edges and fine parts, S/N 
ratio, resolution and the like, to those provided by the linear interpolation or by 
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classification-adaptation process wherein the R, G and B signals are classified 
independently. 
[0076] 

It is proposed that the pixel data corresponding to each G pixel that has 
more data than any other pixel of the color filter array be used in calculation, 
thereby to enhance the precision of predicting R pixels and B pixels. (See, for 
example, Jpn. Pat. Appln. KOKAI Publication No. 11-082228.) The present 
embodiment of this invention can provide images of quality equal to or higher than 
that of the images generated by the method proposed in the publication. 
[0077] 

The structures of the prediction tap and class tap are not limited to those 
shown in FIGS. 10 and 11. The structures of the prediction tap and class tap may 
be changed, if necessary, in accordance with various conditions, such as the 
arrangement of the color filters or complement-color filters and the resolution of 
the images to be generated. For example, more pixels may be extracted than 
those constituting a class tap or a prediction tap if the output image needs to have 
higher quality. 
[0078] 

An embodiment of the invention is a digital camera. Nonetheless, the 
present invention can be applied to video apparatuses such as VTRs, each 
incorporating a camera, and also image-processing apparatuses for use in broadcast 
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stations. 
[0079] 

[Advantageous Effect of the Invention] 

In the present invention, the ADRC process is effected on each of the color 
signals that have passed the color filter array, have been extracted and are now 
contained in a class tap. The relation between the color signals is added to the 
result of the ADRC process. The present invention can therefore perform an 
interpolation process with higher accuracy than otherwise and can, hence, generate 
image data representing an image of high resolution. 
[Brief Description of the Drawings] 

[FIG. 1] 

A block diagram showing the entire structure of an embodiment of the 
present invention. 
[FIG. 2] 

A flowchart explaining the overall process an embodiment performs. 
[FIG. 3] 

A block diagram illustrating an example of a section of an embodiment of 
the invention. 
[FIG. 4] 

A block diagram explaining the prediction or inference effected in the 
ordinary classification-adaptation process. 
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[FIG. 5] 

A block diagram explaining a device that performs learning in the ordinary 
classification-adaptation process. 
[FIG. 6] 

A block diagram showing a section of the device that performs learning an 
embodiment of this invention. 
[FIG. 7] 

A flowchart explaining a part of the process an embodiment of the 
invention 
carries out. 
[FIG. 8] 

A flowchart explaining the learning performed in an embodiment of this 
invention. 
[FIG. 9] 

A diagram showing an example of the arrangement of a color filter array to 
which this invention can be applied. 
[FIG. 10] 

A diagram illustrating an example of a tap configuration used in an 
embodiment of the invention. 
[FIG. 11] 

A diagram depicting another example of a tap configuration used in an 
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embodiment of the invention. 
[FIG. 12] 

A diagram explaining the problem with the conventional method of 
processing images. 

[Explanation of Reference Numerals] 
2 CCD ; 5 image-signal processing circuit ; 31 interpolation section ; 25 ADRC 
circuit ; 26 classification process circuit ; 27 adaptation process circuit ; 28 
coefficient memory ; 49 coefficient memory 
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[Document Name] ABSTRACT 
[Abstract] 

[Problems to be Solved] 

To enhance the accuracy of interpolating an output of a single-plate CCD, 
thereby to generate image data equivalent to the data generated by a three-plate 
CCD. 

[Means to Solve the Problems] 

An image obtained by performing a prescribed process on an output of a 
single-plate CCD is supplied to an ADRC circuit 25. The ADRC circuit 25 
extracts, as a class tap, the pixels at the positions corresponding to a plurality of 
color signals. In accordance with the data of the class tap, the circuit 25 performs 
an ADRC process on each color signal, thereby generating a re-quantized code. 
A classification process circuit 26 determines a class from the re-quantized code 
and the relation between the color signals. An adaptation process circuit 27 reads, 
from a coefficient memory 28, the prediction coefficients corresponding to the 
class determined. The circuit 27 generates image data which represents pixels, 
each having all color components, and which therefore equivalent to one generated 
by a three-plate CCD. 
[Selected Figure] FIG. 3 
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